Soft robots have continuum solid bodies that can deform in an infinite number of ways. Controlling soft robots is very challenging as there are no closed form solutions. We present a learning-in-the-loop co-optimization algorithm in which a latent state representation is learned as the robot figures out how to solve the task. Our solution marries hybrid particle-grid-based simulation with deep, variational convolutional autoencoder architectures that can capture salient features of robot dynamics with high efficacy. We demonstrate our dynamics-aware feature learning algorithm on both 2D and 3D soft robots, and show that it is more robust and faster converging than the dynamics-oblivious baseline. We validate the behavior of our algorithm with visualizations of the learned representation.

Summary The paper proposes a differentiable pipeline that can jointly learn a latent space representation (via variational autoencoder) for controlling soft robots and optimize for the controller and the soft robot parameters for tasks in simulation, such as making a soft 2D robot walk forward as fast as possible. The work is made possible by using a differentiable hybrid-particle-grid based soft material physics simulator. The authors provided insightful details on the alternative minimization scheme for training the autoencoder, the controller neural network, and the robot parameters in tandem. The proposed framework was evaluated on 5 simulated experiments that show controllers using the learned representation outperforms ones using the baseline representation obtained via k-means clustering. Review While the performance of the system is impressive, the motivation of the approach is not well-communicated in 3 folds: In discussing the proposed hybrid-particle-grid based soft robot representation vs finite element methods, the authors claim that the high "degrees of freedom of finite element methods is impractical for most modern control algorithms."

This is a borderline paper showing how to learn a latent representation using a VAE for control of soft robots. There were concerns by the reviewers about how the results could be generalized and scaled to other simulations and systems. However, there was consensus that the work was novel and should be presented at NeurIPS.

Soft robots have continuum solid bodies that can deform in an infinite number of ways. Controlling soft robots is very challenging as there are no closed form solutions. We present a learning-in-the-loop co-optimization algorithm in which a latent state representation is learned as the robot figures out how to solve the task. Our solution marries hybrid particle-grid-based simulation with deep, variational convolutional autoencoder architectures that can capture salient features of robot dynamics with high efficacy. We demonstrate our dynamics-aware feature learning algorithm on both 2D and 3D soft robots, and show that it is more robust and faster converging than the dynamics-oblivious baseline.